The Adoption of POUE and OUE of IT Investments in the Jordanian Business Environment

The process of IT projects evaluation often seems to cease just as quantifiable results start to become available – in Operational Use evaluation (OUe). This paper investigates OUe of IT projects and it’s contrast it with the evaluation undertaken during the specification, construction, and testing of IT projects. We decided to call such technique Prior Operational Use evaluation (POUe) to distinguish it from OUe. The analysis of 41 usable responses from 106 companies in the Jordanian business environment shows that many companies appear not to undertake OUe evaluation. However, where OUe evaluation was conducted, it appears to be of clear value to organizations. Results suggest that most decision makers do not place much importance on OUe in their IT projects. Most managers tend to think of OUe technique only as a formality rather than a proper evaluation process. INTRODUCTION Expenditure on Information Technology (IT) in developed countries -and other developing countries for that matteris continuously increasing as public and private sectors rely more and more on IT (Al-Yaseen et al., 2006). Accordingly, the issue of assessing the value of IT is increasingly a concern for all decision makers. Recently, a large percentage of organizational new capital investment is spent on IT, directly or indirectly. Justifying expenditure on IT is a long standing problem, and managers for the past few decades have expressed concerns about the value they are getting from IT investments; moreover they have been searching for ways to evaluate and justify the use of IT. Such a continuous increase in investment coupled by continuous need for justification presents a challenge to the information systems community. Al-Yaseen, H., El-Haddadeh, R., Al-Jaghoub, S. and Al-Gweri, A. THE ADOPTION OF POUE AND OUE OF IT INVESTMENTS 2 Many organizations reported that they are uncertain about how to measure the impact and the outcomes of their IT investments. This is mainly attributable to the fact that IT returns-on-investment are mostly intangible which makes it difficult to measure using traditional accounting practice. Many authors agree that evaluation of investment is a key issue for such IT projects and their management: Kumar (1990); Dabrowska and Cornford (2001); and Irani et al. (2002). Investment justification and evaluation of effectiveness is traditionally – within fields other than IT – a complex process. IT evaluation has been a widely explored issue in order to resolve the above issues and in search of reliable measurement drivers. Most of the theoretical literature in IT evaluation {such as Bradford and Florin (2003); Gunasekaran et al., (2001); Lin and Pervan (2003); Liu et al. (2003); Remenyi et al. (2000); and Irani and Love (2002)} tends to depart from the traditional accounting-based evaluation methods by appreciating the intangible aspects of IT benefits as well as the tangible ones. Authors are more inclined to view evaluation as part of the planning activity only or, in some cases, as part of the development process. There are also a number of empirical studies – such as those reviewed by (Ballantine et al, (1996) – which examined ex-ante evaluation, yet only a few {for example Kumar (1990) and to some extent Beynon-Davis et al., 2004)} that have explored the expost evaluation. Generally speaking most empirical and theoretical articles (with very few exceptions) tend to classify IT evaluation as a planning activity or take a temporal view along the development life-cycle only to stop short of the operational phase. Although a number of the above authors have touched upon this phase, evaluation activities are still not represented as integral parts of the evaluation process. The extent to which organizations adopt rigorous evaluation at the operational phase is unknown. In this paper, we aim to empirically explore IT evaluation process by extending the temporal view – with more concentration on the operational phase – in order to understand issues related to IT evaluation after project completion. We start in the following section by defining IT evaluation for the purpose of this research. We then use this as the a theoretical basis for the collection of data from major companies in Jordan regarding their approaches and processes for IT project evaluation, as well as their rationale and application of any Operational Use Evaluation (OUe) that they conducted. The section after that redefines the research problem and the key research questions in relation to the two forms of evaluation. The next sections discuss the research methodology, data collection, results, and synthesis; respectively. In the final section we present lessons learned from this research. THE PURPOSES AND FORMS OF EVALUATION Evaluation has been defined as the process of assessing the worth of something (Beynon-Davis et al., 2000). Another definition given is that it can be defined as the process of establishing – by quantitative or qualitative means – the worth of IT to the organisation (Willcocks, 1992). We take the stance that evaluation is a process that takes place at different points in time, or continuously, explicitly searching for (quantitatively or qualitatively) the impact of IT projects (Eldabi et al., 2003; Al-Yaseen et al., 2006). The value of this latter definition is that it explicitly recognises the different stages in the full lifecycle of an Information System in which evaluation is performed, and provides the opportunity to discriminate between two decidedly different views of the evaluation process, each serving different aims. Al-Yaseen, H., El-Haddadeh, R., Al-Jaghoub, S. and Al-Gweri, A. THE ADOPTION OF POUE AND OUE OF IT INVESTMENTS 3 The first view of evaluation is as a means to gain direction in IT project. Here, ‘predictive’ evaluation is performed to forecast the impact of the project. Using financial and other quantitative estimates, the evaluation process provides support and justification for the investment through the forecasting of projected baseline indicators such as Payback, Net Present Value (NPV) or Internal Rate of Return (IRR) (Farbey et al., 1993; Liu et al., 2003; Yeo and Qiu, 2003). It is known variously as ‘ex-ante’ evaluation (Remenyi et al., 2000; Love et al., 2005; Jagtman et al., 2007; Oikonomou et al., 2007), ‘formative’ evaluation (Brown and Kiernan, 2001; Tao et al., 2006; Sholtz et al., 2006; Ballantyne and Hughes, 2006), or as we shall refer to it, ‘Prior Operational Use evaluation’ (POUe). This form of evaluation guides the project, and may lead to changes in the way the system is structured and carried out. It does not however give any feedback beyond the design, implementation, and delivery of the project outcomes. In contrast, evaluation can also be considered in terms of the effectiveness of the IT system in situ – what a system actually accomplishes in relation to its stated goals (Eldabi et al., 2003; Al-Yaseen et al., 2004). This form of evaluation draws on real rather than projected data, and can be used to justify adoption (Love and Irani, 2001; Irani, 2002); estimate the direct cost of the system, estimate the tangible benefits of the system (Liu et al., 2003); ensure that the system meets requirements (Irani, 2002); measure the system effectiveness and efficiency (Poon and Wagner, 2001); measure the quality of programs and to estimate indirect costs and other costs (Love and Irani, 2001); or to measure the quality of programmes (Eldabi et al., 2003). This type of evaluation should be performed during the operational phase of the project. It is known variously as ‘ex-post'’ evaluation (Remenyi et al., 2000; Kumar, 1990; Beynon-Davis et al., 2004; Van Den Bergh et al., 2007), ‘summative' evaluation (Berry and Linsley, 2006; Schwartz and Mayne, 2005) .We shall refer to this type as ‘Operational Use evaluation’ (OUe). Figure 1 shows these forms of evaluation with respect to the life cycle from a system’s inception to the end of its useful life. Figure 1. IT evaluation forms in the systems’ life cycle THE PROBLEM AND THE RESEARCH OPPORTUNITY Most of the literature {such as Beynon-Davis et al. (2000); Farbey et al. (1999); Jones and Hughes (2000); Walsham (1999); Remenyi et al. (2000)} attempts to improve the process of evaluation by means of either a) consolidating and enumerating more factors to consider in the evaluation, or b) adding more theoretical rigour to the techniques used (Irani, 2002; Irani and Love, 2002). As mentioned above, most studies are concentrated on what we termed the POUe phase with high emphasis on early stages of development. In contrast, we find that only rarely that OUe has been studied. Prior Operational Use Evaluation Operational Use Evaluation Development stages System into operational use Time Systems’ life cycle Al-Yaseen, H., El-Haddadeh, R., Al-Jaghoub, S. and Al-Gweri, A. THE ADOPTION OF POUE AND OUE OF IT INVESTMENTS 4 The main problem is that there is no body of knowledge in the area to help improve the techniques used in evaluation at this stage, which encourages decision makers to refrain from employing it altogether. For this reason we have decided to research into the practitioners’ perceptions of the evaluation process and the practices associated with the evaluation adopted within large organisations. We attempt to obtain insights into OU evaluation in order to identify the real extent to which OU evaluation is practised and what lessons that could be learned to improve knowledge about it. To do that – we believe – the following questions need to be answered by practitioners who are most involved with the evaluation processes. Such answers are obtained by posing the following questions as a platform for our research activity: • The extent to which POUe and OUe of IT are practiced? • What are the main reasons for adopting POUe and OUe of IT? • What are the evaluation method used when adopting POUe and OUe of IT? • Who is involved in the evaluation process throughout the development stage of the system? RESEARCH APPROACH As suggested by the argument of Tashakkori and Teddlie (2003), we opted for quantitative research through questionnaires as an appropriate instrument base for starting the research. No doubt other research approaches would be beneficial and we anticipate other researchers might follow up on this. The following section describes the processes of questionnaire design, deployment, and analysis used, and summarizes the participant characteristics. RESEARCH PHASES Phase one reviews both types of evaluation (POUe and OUe). The main issues identified in the literature were then used to develop a questionnaire that focuses on how organizations carry out evaluation of their IT systems. The questionnaire is split into six sections centered on gathering information on: • Organizational background. • Information technology infrastructure. • Business issues of IT investment. • Prior Operational Use evaluation in different stages of system life cycle (feasibility, design, implementation, and testing and completion). • Operational Use evaluation, as well as • Other information related to both types of evaluation. Before the formal survey was sent to the companies, two pilot iterations were conducted. The first iteration involved two doctoral students. Based on their feedback, certain items in the questionnaire were modified, along with minor layout changes which were made in order to improve clarity and readability. The second iteration involved five professionals – three academics, one IT manager in a business organization, and one business analyst in another organization. There were only cosmetic changes at this iteration, giving us the confidence to issue the questionnaire. In phase two, the questionnaire developed in phase one was sent to the top 106 organizations in Jordan. The questionnaires were mailed to IT managers or top executives. As shown in Table 1, returns covered a variety of organizations from financial services, information technology, manufacturing, transport, central government, Al-Yaseen, H., El-Haddadeh, R., Al-Jaghoub, S. and Al-Gweri, A. THE ADOPTION OF POUE AND OUE OF IT INVESTMENTS 5 consultancy, retail/wholesaling, and publishing. Of the 106 questionnaires posted, 45 responses were received; 3 were returned unanswered and 2 were returned but incomplete. The latter two categories of responses were ignored making the final number of usable responses 41, giving a response rate of 38.7%. This rate was considered to be above expectation given that the generally accepted average responses to non-incentive based questionnaires are around 20%. Table 1: Organisations in the sample Organisation/sector Number Government sector 20 Banking sector 14 Insurance sector 8 Services sector 25 Industrial sector 39 Total 106 In phase three, we analyzed the data from the responses of the questionnaire using a combination of the parametric statistical methods, Descriptive Analysis and Factor Analysis (Pett, et al, 2003). Organizations were asked to select from the list the closest choice the reasons of adopting POUe and OUe. A summary of the key responses to the questionnaire – the reasons for adopting Prior Operational Use evaluation (codename: POUeR), and Operational Use evaluation (codename: OUeR) – are tabulated in Appendix 1. Each of these variables were measured using a five point Likert scales (1 = not important and 5 = very important). For technically interested readers we report that a factor analysis technique was employed in order to identify possible categories. Factor analysis was performed in three steps (following Berthold and Hand, 2003): • A matrix of correlation coefficients for all possible pairings of the variables was generated. • Factors were then extracted from the correlation matrix using principal factors analysis. • The factors were rotated to maximize the relationships between the variables and some of the factors and minimize association with others using Varimax Kaiser Normalization, which maintained independence among the mathematical factors. The Eigenvalues determined which factors remained in the analysis. Following Kaiser’s criterion, factors with an Eigenvalue of less than 0.5 were excluded. A Screen plot provides a graphic image of the Eigenvalue for each component extracted. RESPONDENTS’ CHARACTERISTICS The average annually IT budget for the organizations in the sample was J.D. 1,500,000 ($ 2,100,000). 17% of the participating organizations have annually IT budget exceeding J.D. 3,315,000 ($ 4,650,000). On average, the participating organizations had been using IT for approximately 10-20 years and most had a history of more than 10 years of using IT. 92% of the participating organizations had a central integrated IT infrastructure department, while 8% of each department in the participating organizations had its own IT infrastructure. 36.2% of the participating organizations had adopted IT as a response to problem(s); whilst 8.6% had adopted IT searching for ways of improving effectiveness and standing in the marketplace, and 55.2% had adopted IT for both reasons. DATA ANALYSIS AND PRELIMINARY FINDINGS This section presents aggregated results from direct answers to the research questions mentioned above. The basic issues considered here are: reasons for adopting either types of evaluations and criteria for evaluations. Al-Yaseen, H., El-Haddadeh, R., Al-Jaghoub, S. and Al-Gweri, A. THE ADOPTION OF POUE AND OUE OF IT INVESTMENTS 6 All of the responding organizations have and do carry out formal POUe for their IT projects, but only about a third (34.1%) currently perform a formal OUe evaluation. This means that about two third (65.9%) of the organizations do not gather any evidence to establish how successful their IT projects were therefore cannot use such information from OUe evaluation to improve their evaluation techniques or process. REASONS FOR ADOPTING POUE The results are presented in Table 2. Using a factor analysis cut-off level of 0.5, four factors were considered the main reasons of adopting POUe (see Figure 2), which we describe as ‘system completion and justification’, ‘system costs’, ‘system benefits’, and ‘other reasons’. The first factor ‘System completion and justification’ is highly correlated with seven variables, the second factor ‘system costs’ is highly correlated with eleven variables, and the third factor ‘system benefits’ are highly correlated with four factors, whilst the fourth factor ‘other reasons’ is highly correlated with two variables barriers for adopting the system which was also found to be the least evaluated reason in practice, as shown in Table 2. A glossary of variables is found in Appendix 1. Table 2: Reasons for adopting Prior Operational Use evaluation – Factor analysis